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Prospects of measuring sin 2 2(9i3 and the sign of Z\m 2 with a 
massive magnetized detector for atmospheric neutrinos 

T. Tabarelli de Fatis 

INFN, Sezione di Milano, Piazza della Scienza 3, 1-20126 Milan, Italy 
February 22, 2002 

Abstract. The pattern of oscillation parameters emerging from current experimental data can be further 
elucidated by the observation of matter effects. In contrast to planned experiments with conventional 
neutrino beams, atmospheric neutrinos offer the possibility to search for Earth-induced matter effects with 
very long baselines. Resonant matter effects are asymmetric on neutrinos and anti-neutrinos, depending 
on the sign of Am 2 . In a three-generation oscillation scenario, this gives access to the mass hierarchy of 
neutrinos, while the size of the asymmetry would measure the admixture of electron neutrinos to muon/tau 
neutrino oscillations (the mixing angle ©13). The sensitivity to these effects is discussed after the detailed 
simulation of a realistic experiment based on a massive detector for atmospheric neutrinos with charge 
identification. We show how a detector, which measure and distinguish between and charged current 
events, might be sensitive to matter effects using atmospheric neutrinos, provided the mixing angle 613 is 
large enough. 
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1 Introduction 

The very long baselines available with atmospheric neutri- 
nos offer the possibility to search for Earth-induced matter 
effects. In that endeavour, atmospheric neutrino experi- 
ments are not contested by current and planned acceler- 
ator beam experiments - whose baselines are too short 
for a significant effect - and offer the opportunity to test 
the neutrino mass hierarchy. The observation of matter 
effects might also be possible with upgraded conventional 
neutrino beams jy and at neutrino factories (2) . 

Matter effects can play an important role if there are 
significant contributions of v e or Sterile to atmospheric 
neutrino oscillations. The non-observation of large matter 
effects has been already exploited by Super-Kamiokande 
to exclude a large contribution of — Sterile oscillations 
to atmospheric neutrinos |^,^|]. For a contribution of non- 
maximal — ^sterile oscillations, matter effects would also 
manifest themselves in differences in the oscillation pat- 
terns for neutrinos and anti-neutrinos. These differences 
could be measured with a magnetized detector with the 
capability to identify the muon charge H. 

Matter effects could be detectable even in standard 
three flavour oscillation scenarios Q. A sub-dominant v e 
mixing could sizeably modify the transition probability 
in particular regions of phase space where the — ► u e 
transition becomes resonant in matter. Depending on the 
sign of Am 2 , these effects occur either for neutrinos or for 
anti-neutrinos only. By comparing the neutrino and anti- 
neutrino distributions, the sign of Am 2 , and therefore the 



neutrino mass hierarchy, could be determined if a signal 
were observed. The size of this effect would measure the 
admixture of electron neutrinos. 

In this work, the sensitivity to matter effects of a mas- 
sive magnetized detector for atmospheric neutrinos is dis- 
cussed in the framework of a three flavour scenario with 
one mass scale dominance for atmospheric neutrinos (i.e. 



Am 2 



''1.2) 



The full simulation of the MONO- 



LITH detector |j is used for a realistic description of 
resolution and reconstruction effects. About 200 kty of 
MONOLITH exposure (6 y) will be sufficient to explore 
regions not yet ruled out by the bound on v e — > v x oscil- 
lations derived by CHOOZ results 0). The potential sen- 
sitivity of a detector (or array of detectors) with masses 
significantly larger than MONOLITH will be described. 



2 Three neutrino oscillations with one mass 
scale dominance for atmospheric neutrinos 

The current phenomenology of neutrino oscillation can- 
not be reconciled with a three-neutrino oscillation sce- 
nario, unless one of the experimental evidences for neu- 
trino flavour conversion is disregarded)^. In the "one mass 
scale dominance" for atmospheric neutrino scenario 
the choice is made to disregard the LSND result || and 
describe the disappearance observed with atmospheric 
neutrinos as oscillations of into v T , which are (almost) 

Technically oscillations have not been observed yet. 
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pure admixtures of two mass eigenstates (say v-i and v 3 ) 



with Ar 



'2:; 



= Ar 



l Ati 



Solar neutrino involve oscillations 



between z/i and z/2 with Am{ 2 = Am,g un << Am Atm . 

In this scheme 
of atmospheric neutrinos (i.e. Ar 



Am 2 Sun is too small to affect oscillation 



l Sun 



L/E << 1) and can 



be approximated to zero. In this limit and up to terms 
proportional to the identity, the Hamiltonian for the time 
evolution of neutrinos propagating in matter of constant 
density reads: 



H 




where Am 2 — Am 2 3 ~ Am 2 3 , ^/2GpN e is the effective 
potential for the scattering of electron neutrinos off the 
external field due to electrons of volume density N e and 
U is the flavour mixing matrix in vacuum^. 

For a given baseline L and constant matter density, 



with 



v(L) = S(L)v(0) 



S(L) = U 



(2) 



(3) 



V 



where Am describes the energy-level spacings in matter 
and U the effective mixings. The transition amplitudes for 
v a — > i/p are given by S matrix elements: 



A(a -> 13; L) = S(L) 0C 



(4) 



from which the transition probabilities may be calculated. 
The exact expressions for the effective mass differences 
and mixings in matter of constant density after diagonal- 
ization of the Hamiltonian of equation ([!]) are given in 

Equation (lh implies that the (1,2) sector is inopera- 
tive in atmospheric neutrinos, the CP phase of the mixing 
matrix is unobservable and the oscillations are entirely de- 
scribed by three parameters: 6*13, 6*23 and Am 2 . The pa- 
rameter 6*i3 measures the admixture of electron neutrinos 
to atmospheric neutrino oscillations and is related to U e s 
of the mixing matrix by | f/ e 3 1 2 = sin 2 6*13. This parameter 
is bound to be small by CHOOZ results (sin 2 20 13 < 0.1 
at 90% C.L. @ for large Am 2 ), but the (1,3) transition 
can become resonant in matter and sizeably modify the 
oscillation probabilities of electron and muon neutrinos. 

In a medium of constant density, the resonant energy 
and the resonance width are given by 



r, 



R 



± cos 26*13 
2 sin 26>i3 



Am 2 
2V2G F N e ' 



(5) 



(6) 



The corresponding Hamiltonian for anti-neutrinos is ob- 
tained by changing U — > U* and the sign of the effective po- 
tential. 
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Fig. 1. Electron density profile scaled by 2\/2Gf in the Earth 
according to the Preliminary Reference Earth Model jnj and 
the discrete approximation adopted in this study. 

where the + (-) sign in eq. (^) is referred to neutrinos 
(anti- neutrinos). Thus the resonance occurs either for neu- 
trinos or for anti-neutrinos only, depending on the sign of 
Am 2 . In the limit of small mixing angles, the resonance 
width shrinks like sin2(9i3 and the effect will eventually 
become unobservable when sharper than the experiment 
resolution. 

On the other hand, the resonant energy is practically 
independent of 6*13 and only depends on the mass differ- 
ence of neutrino states and on the medium density. Thus, 
the position of the resonance can be predicted accurately 
when Am 2 is known. For Am 2 = 0.003 eV 2 , the resonant 
energy is around 3 GeV, 7 GeV and 10 GeV in the Earth 
core, mantle and external mantle respectively, well within 
the MONOLITH acceptance (see below). These values 
give only a qualitative indication of the energy region of 
interest for matter effects with atmospheric neutrinos. A 
more precise prediction of the effects requires that the ex- 
act density profile along the neutrino path be considered. 

The explicit computation of transition probabilities 
has been addressed numerically. The Earth density pro- 
file of the Preliminary Reference Earth Model |l(J has 
been approximated with eleven shells of constant density 
(see Fig. m) and neutrino propagation has been computed 
from the (time ordered) product of evolution operators in 
matter of constant density: 

v(L) = S(L)v(Q) 

= S„(i„)5 n _i(L„_i)... Si(Li)v(0). (7) 

where Si(Li) describes the neutrino propagation along the 
path Li in the i-th shell and n is the total number of shells 
crossed, which both depend on the neutrino angle. 

The — ► and v e — > conversion probabilities as 
a function of L/E and cos 9 Nadir are shown in figure || for 
Am 2 = +0.003 eV 2 and sin 2 20 13 = 0.1. These values of 
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Fig. 2. In the upper and middle panels, the conversion proba- 
bilities for atmospheric neutrinos crossing the Earth are shown 
as a function of L/E and cos0Nadir for Am 2 — +0.003 eV 2 and 
sin 2 (2(9i3) = 0.1. The probability for T> e — > conversion is 
amplified by one order of magnitude to make it visible on the 
same scale as the other conversions. The lower panels shows 
the probability of observing muon neutrinos considering both 
v ^ — > and v e — > Vp conversions and the neutrino fluxes. 



oscillation parameters sit at the border of the 90% C.L. re- 
gion excluded by CHOOZ. As a positive sign for Am 2 was 
chosen, the L/E pattern typical of vacuum oscillation is 
practically unaffected by matter effects for anti-neutrinos, 
while it is clearly modified for neutrinos. For neutrinos 
crossing the Earth's core (i.e. cos > 0.85), sharp 

structures appear in the oscillation pattern. For lower val- 
ues of cos 8 Nadir, the distortion with respect to the vac- 
uum L/E pattern for muon neutrinos is broader and most 
relevant around the first maximum, with a suppression of 
about 50%, and the second minimum of the conversion 
probability. Electron neutrino oscillations into muon neu- 
trinos are correspondingly enabled in the same region of 
the L/E and cos 9 Nadir plane. The energy and angular 
regions of interest for these effects range from 4 GeV to 
15 GeV and from 40° to 60°, corresponding to a baseline 
of about 9000 km. 

For smaller values of the mixing angle 6*13 , the energy 
and angular positions of this distortion are almost inde- 



pendent of sin 2 26>i3, but the distortion becomes sharper, 
in agreement with the expectations for neutrino propaga- 
tion through a constant density medium (like the Earth's 
mantle in first approximation). 

The MONOLITH detector (see below) is optimized to 
reconstruct muon neutrinos and anti-neutrinos, the ex- 
pected distribution of which is a combination of the con- 
version probabilities for Vp — > Vp (P w ) and v e — > Vp 
(Pep) weighted by the corresponding neutrino fluxes. The 
probability of observing muon neutrinos is thus given by 
Ppp + l/ r Pep, where r is the Vp to v e flux ratio. Over 
the region where relevant matter effects occur, r ranges 
between 2 and 3 and the disappearance of muon neutri- 
nos through matter effects is only partly compensated by 
Ve. — * fp transitions. This is shown in the lower panels of 
the figure for both neutrinos and anti-neutrinos: the flux 
suppression with respect to pure Vp-v T oscillations is still 
about 30% around the first maximum of the reappear- 
ance probability. The detailed simulation is addressed in 
the following. 

3 Experiment simulation 

The differential distribution of the atmospheric neutrino 
fluxes at Gran Sasso has been generated according to the 
predictions of the 1-D Bartol model |ll| . Neutrino inter- 
actions have been calculated with GRV94 parton distri- 
butions Jl2]] with explicit inclusion of the contribution of 
quasi-elastic scattering and of single pion production to 
the neutrino cross-sections 

Events have been processed through the full simulation 
and reconstruction programs of the MONOLITH detector 
described in || . The proposed detector for MONOLITH is 
a massive tracking calorimeter with a coarse structure and 
intense magnetic field. The detector has a large modular 
structure (figure ||). One module consists in a stack of 120 
horizontal 8 cm thick iron planes with a surface area of 
14.5 x 15 m 2 , interleaved with 2 cm planes of sensitive 
elements. The total mass of the detector for two modules 
is about 34 kt. The magnetic field configuration is also 
shown in figure |3| iron plates are magnetized at a magnetic 
induction of rs 1.3 T. 

The active elements (Resistive plate chambers with 
glass electrodes O]) provide two coordinates with a pitch 
of 3 cm and a time resolution of order 1 ns 

After reconstruction, some minimal requests have been 
applied to select a pure sample of muon neutrino charged 
current interactions: 

— a muon candidate firing at least 7 layers is required; 

— the muon energy must exceed 1.5 GeV; 

— the event is required to be cither fully contained in 
a fiducial volume defined by requiring no hits in the 
first/last 4 layers and no hits in the first/last 10 strips 
in X and Y, or to have a single outgoing track (muon) 
with a reconstructed range greater than 4 metres. 

— the L/E resolution of the event, determined from the 
angular and energy resolution and estimated from the 
event kinematic, is required to be better than 50% 
FWHM. 
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Fig. 3. Schematic view of the MONOLITH detector. The ar- 
rangement of the magnetic field is also shown. 



These selections guarantee a relatively high efficiency for 
an adequate resolution in the reconstruction of the neu- 
trino energy and direction in the region of interest for 
matter effects. They result in an effective threshold around 
3 GeV of neutrino energy and in a fairly constant efficiency 
between 50% and 55% from 5 GeV onwards. The neutrino 
angle is estimated from the muon direction with a reso- 
lution of about 15 degrees at 3 GeV and then improving 
with a -E~ ' 5 power law. The neutrino energy, estimated 
from the muon and hadron energy, shows a resolution of 
about 15-20% in the energy region where resonant effects 
are expected (4-15 GeV), smoothly broadening to 25% at 
very high energies, reflecting the worsening of the muon 
momentum resolution from track curvature. Charge as- 
signment is correct in more than 95% of the cases, with- 
out additional requests on the track fit quality. The back- 
ground due to internal and external sources is estimated 
to be negligible. More details on the MONOLITH perfor- 
mance can be found in || . 

Only internal events are selected and, in the energy re- 
gion of interest for resonant effects, about 75% of these are 
fully contained in the detector. External events (up-going 
muons from neutrino interactions in the rock below the 
detector) are of little use in this study. Their average neu- 
trino energy is much higher than the resonant energy and 
the resolution in the reconstruction of the neutrino energy 
is insufficient to resolve the typical resonance width. 



4 Analysis and results 

Although matter effects make the L/E variable not suffi- 
cient for oscillation studies, an optimized analysis can be 
performed in the L/E and cos 9 Nadir plane. The energy of 
the resonance and the oscillation phase in vacuum scale 
both with Am 2 and the main distortion to the vacuum 
oscillation pattern always occurs around the first maxi- 
mum of the reappearance probability. As visible in figure 
^, the L/E pattern is only marginally modified over the 
first oscillation period by matter effects. This difference is 
even less evident when resolution and detector efficiency 
effects are considered (see Fig. ||). For Am 2 = +0.003 eV 2 
and the largest values of sin 2 20i 3 allowed by CHOOZ re- 
sults, the distortion on the L/E pattern (integrated over 




L/E (km/GeV) 



Fig. 4. Left: L/E spectra for up-going (hatched his- 
togram) and down-going muon neutrino (top) and muon anti- 
neutrino (bottom) events for three neutrino oscillations Am 2 — 
+0.003 eV 2 , sin 2 26>23 = L and sin 2 26>i 3 = 0.1. Right: Ratio 
of the up-going to down-going L/E distributions. The curve 
represents the expectations for pure — > v T oscillations. The 
experiment has been simulated with high statistics. Event rates 
and error bars are normalized to 6 y of MONOLITH exposure 
(200 kty). 



all neutrino baselines) is barely visible and located be- 
yond the first minimum in the survival probability. As 
demonstarated in figure |^, this distortion is mainly due to 
neutrinos crossing the Earth's mantle and in any case no 
distortions are predicted over the first half-period of oscil- 
lation. It follows that the measurement of the oscillation 
frequency from the position of the first minimum in the 
L /E pattern is marginally affected by the presence of mat- 
ter effects. The analysis of the L/E pattern, assuming that 
no matter effects are present, will therefore precisely mea- 
sure Am 2 and tell where such effects have to be searched 
for in a two dimensional analysis (bin optimization). 

The analysis of simulated data has been based on a 
binned maximum likelihood fit. The expected L/E distri- 
butions for neutrino and anti-neutrino events have been 
fitted to the reconstructed data from simulation in five 
cos9]\radir bins of width 0.2. The binning in L/E has been 
tuned according to the estimated value of Am 2 from a 
preliminary fit of the L/E distribution. 

The expected number of events in each bin is a function 
of the oscillation parameters (O13, O23, Am 2 ) and of the 
neutrino fluxes of all the flavours. In the real experiment, 
down-going neutrinos will constitute a reference sample 
of "unoscillated" neutrinos to check the muon neutrino 
and anti-neutrino rates. The rate of muon neutrinos orig- 
inating from electron neutrino conversions will have to be 
predicted by flux calculations of electron neutrinos. These 
predictions will be constrained both by the measurement 
of the correlated rate of down-going muon neutrinos in 
the same detector and by the direct measurement of the 
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Fig. 5. L/E spectra for up-going neutrinos crossing the 
Earth's core (left) and mantle (middle and right) assuming 
Am 2 = +0.003 eV 2 , sin 2 2023 = 1. and sin 2 26>i 3 = 0.1 (dots) 
compared to the expectations for pure — * v T oscillations 
(histogram). The experiment has been simulated with high 
statistics. The error bars, shown only in the region where mat- 
ter effects are sizeable, are normalized to 6 y of MONOLITH 
exposure (200 kty). 



electron neutrino to muon neutrino flux ratio in other ex- 
periments (Super-Kamiokande). A detailed study to quan- 
tify the related amount of systematic uncertainty has not 
been performed and an overall uncertainty of 10% in the 
predicted rates of neutrinos and anti-neutrinos has been 
assumed for a preliminary estimate of the experiment sen- 
sitivity. 

The likelihood function has been defined as: 



In£ = lnJJ 



i. q 



e- A ^(A qfH>q ) 



1 {A q I? 



(8) 

where the subscripts i and q are referred to bins in the 
(cos 9 Nadir, L / E) plane and of muon charge respectively; 
Ui t q and A q [ii A are the observed and expected number of 
up-going neutrino events in the i, q-th bin and the last 
term in the likelihood accounts for flux, cross-section and 
selection efficiency uncertainties for neutrinos and anti- 
neutrinos separately. 

Two different analyses have been performed to deter- 
mine the sensitivity to the mixing parameter and to the 
sign of Am 2 . 



4.1 Sensitivity to sin 2 20 13 

In this analysis it has been assumed that the mixing in the 
(2,3) sector be maximal (sin 2 26>23 = !)■ This corresponds 
to the best current estimate from Super-Kamiokande data. 
Individual experiments of high statistics have been gener- 
ated for different values of sin 2 26>i3 an d Am 2 and tested 
with a likelihood ratio analysis against the pure v^/v T os- 
cillation scenario (i.e. sin 2 26>i3 — 0), assumed as null hy- 
pothesis. From this procedure, the values of the oscillation 



parameters that would give an experimental result incom- 
patible with the null hypothesis have been determined. 
The likelihood ratio is defined by: 



A = 



£(A g ,zW,sin 2 26>23,sin 2 26>i 3 |sin 2 26>i 3 = 0) 
C(Aq, Am 2 , sin 2 26> 23 , sin 2 20 13 ) 



, (9) 



representing the ratio of the maximum likelihood over the 
parameter space of the null hypothesis to the maximum 
likelihood over the entire parameter space. The test statis- 
tics —2 In A asymptotically behaves as a x 2 (l) distribution, 
whence the iso-probability curves have been calculated. 
The expected and observed rates of events in the likeli- 
hood function of equation (j^) have been normalized to 
represent experiments of finite statistics. 

The expected sensitivities (exclusion regions at 90% 
C.L. if no effect is observed) after 200 kty of exposure 
(about 6 y for 34 kt) are shown in Fig. |[ Since the sign 
of Am 2 is not known a priori the curves for both posi- 
tive (continuous line) and negative (dashed line) sign of 
Am 2 are given. The expected exclusion region for posi- 
tive sign of Am 2 is about two times larger than for nega- 
tive sign. This follows from the difference in neutrino and 
anti-neutrino cross sections, fluxes and selection efficien- 
cies, which make the event rate of atmospheric neutrinos 
about two times larger than the rate of anti-neutrinos. 

The sensitivity achievable in a long-term run (or with a 
larger detector), corresponding to an exposure of 400 kty, 
is also shown in the figure. 

For comparison, the region excluded by CHOOZ re- 



sults M and allowed by Super-Kamiokande data 15 



are 



displayed. Also displayed are the expected sensitivities of 
neutrino oscillation programs with conventional neutrino 
beams in the short/medium term. The region covered by 
MONOLITH will be partly accessible to the MINOS ex- 
periment at NuMI |16| and fully accessible to the JHF 



beam to Super-Kamiokande 17 . The latter is expected 



to achieve this sensitivity around 2012 after 5 y of opera- 
tion. 



4.2 Sensitivity to the sign of Am 2 

Experiments at NuMI and JHF search directly for the sub- 
dominant — > v e transition and have too short baselines 
to exploit matter effects. Thus, they cannot measure the 
sign of Am 2 . 

With atmospheric neutrinos, if an effect is observed, 
the sign of Am 2 can also be determined. Assuming no 
prior knowledge on sin 2 26>i 3 , this happens at the 90% 
C.L. on the right-hand side of the dashed curves of figure 
[| irrespectively of the sign of Am 2 , and on the right-hand 
side of the continuous curves, if Am 2 is positive. 

However, as discussed in the previous analysis, pro- 
vided that the mixing sin 2 2(9i 3 be large enough, there is 
a fair chance that it will be measured in the next gen- 
eration of long base-lines experiments. In that case, this 
information can be used to constrain the maximum like- 
lihood fit and and determine the sign of Am 2 from the 
muon/anti-muon rate asymmetry. 
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Fig. 6. Exclusion regions at 90% C.L. if no matter effects are 
observed in MONOLITH after 200 kty (6 y) and 400 kty (12 y) 
of exposure. The curves for both positive (continuous line) and 
negative (dashed line) sign of Am 2 are given. The regions ex- 
cluded by CHOOZ results and allowed by Super-Kamiokande 
data are shown, together with the expected sensitivities of MI- 
NOS (Medium Energy option) and JHF projects (Low Energy 
option). The MINOS sensitivity is practically independent of 
the beam optics configuration in the region of Am 2 allowed by 
Super-Kamiokande data. The JHF sensitivity can be better or 
worse than the one displayed by a factor of two depending on 
the beam optics. 

The likelihood function of eq. (||) can be modified as: 

inr^mr-i^ 22013 -^ 1122013 ^ 2 (10) 

2 cr 13 

In this analysis, sin 2 2023 has been assumed to be max- 
imal and sin 2 2(9i3 has been assumed to be known with 
an accuracy of 30%, representing the observation of a 3tr 
effect at accelerator beams. Similarly to the analysis out- 
lined in the previous section, individual experiments have 
been generated for different values of sin 2 2<9i3 and Am 2 . 
For each (absolute) value of Am 2 , the minimum value of 
sin 2 26>i3 for which the best fit to data with the wrong sign 
of Am 2 is incompatible with data has been determined. 

Figure (upper panel) shows the expected region of 
oscillation parameters over which the sign of Am 2 can be 
determined at the 90% C.L. after 200 kty of MONOLITH 
exposure (i.e. the wrong sign of Am 2 is excluded at the 
90% C.L.). The sensitivity achievable in a long-term run, 
corresponding to an exposure of 400 kty, is also shown. 
For comparison, the region excluded by CHOOZ [Q and 
the region allowed by Super-Kamiokande at 90% C.L. are 
displayed. On the same figure, the regions over which the 
the sign of Am 2 can be determined after 200 kty of expo- 
sure without any prior knowledge of the (1,3) mixing are 
shown for comparison. The 3a discovery limit for the sign 
of Am 2 is shown in the lower panel of the figure. 



> 



E 




Fig. 7. Upper panel: The continuous lines define the boundary 
of the regions of oscillation parameters in the three generation 
scenario over which the sign of Am 2 can be determined at the 
90% C.L. after MONOLITH exposures of 200 kty (6 y) and 
400 kty (12 y), assuming that sin 2 20i3 be known with 30% 
accuracy. For comparison, the broken lines show the regions 
over which the sign of Am 2 can be determined after 200 kty 
exposure assuming no prior knowledge on sin 2 26>i3 regardless 
of the sign of Am 2 (dotted) and for positive Am 2 (dashed). 
The regions excluded by CHOOZ results at 90% C.L. and al- 
lowed by Super-Kamiokande at 90% C.L. data are also shown. 
Lower panel: Regions of oscillation parameters over which the 
sign of Am 2 can be determined at the 3<r level after MONO- 
LITH exposures of 200 kty (6 y) and 400 kty (12 y), assuming 
that sin 2 2©i3 be known with 30% accuracy. 



As expected, the additional constraint on the mixing 
parameter widen the sensitivity region and an exposure of 
200 kty would be sufficient to evade the CHOOZ exclu- 
sion region. This is because in this case one has only to 
discriminate a positive from a negative asymmetry, while 
assuming no prior knowledge of the (1,3) mixing, an asym- 
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metry of any sign should be discriminated from the no- 
asymmetry case. 



5 Systematic uncertainties 

Several sources of systematic uncertainties might affect 
the precision of this measurement. 

The likelihood ratio method described by eq. (9) im- 
plies that no prior knowledge on the values of Am 2 and 
sin 2 2023 is assumed. These parameters can be measured 
in MONOLITH with an accuracy of about 5% with 200 kty 
exposure and reaching 1% for very large exposures. Since 
this would represent a substantial improvement with re- 
spect to the present knowledge, these parameters are not 
constrained in the fitting procedure. 

The limited knowledge of the electron density profile in 
the Earth is also a source of systematic uncertainty. How- 
ever, the sensitivity to matter effects with atmospheric 
neutrinos is dominated by the resonant behaviour of neu- 
trinos crossing the bulk of the Earth's mantle, whose den- 
sity and chemical composition are known with sufficient 
precision. The matter density of the mantle is strongly 
constrained by the knowledge of the Earth's mass and 
moment of inertia, combined with minimal information 
from seismic wave measurements of the density profile. 
The electron density is related to the matter density by 
the Z/A ratio, which is very nearly 1/2 for all reasonable 
chemical compositions of the Earth's mantle. 

As discussed in section ||, the effects related to the lim- 
ited knowledge of neutrinos and anti-neutrino fluxes and 
cross sections have been preliminary accounted for in this 
analysis by assigning an uncertainty of 10% to the pre- 
dicted rates for neutrinos and anti-neutrinos separately. 
For an exact knowledge of these rates, the sensitivity re- 
gions shown in figure |^ would have to be extended by 
about 10%. 

In a real experiment, the sample of down-going neu- 
trino events can be exploited to constrain the neutrino 
and anti-neutrino event rate predictions. The precision of 
this procedure might be expected to scale with the square 
root of the exposure and will not be a fundamental limita- 
tion to the sensitivity to matter effects with atmospheric 
neutrinos. This is further discussed in the next section. 



6 Very large exposures 

The dependence of the experiment sensitivity on the ex- 
posure has been studied. The number of years of MONO- 
LITH exposure has been increased up to represent a col- 
lected statistics of 2 Mty Since selection and reconstruc- 
tion efficiencies are the ones of MONOLITH, this simula- 
tion describes a detector (or array of detectors) of modular 
structure, with each single module comparable to MONO- 
LITH in acceptance and performance. 

Figure ^ shows the expected sensitivity to sin 2 26>i3 as 
a function of the exposure for Am 2 = +0.003. For negative 
sign of Am 2 the sensitivity is about two times worse. 




0.01 1 1 1 1 1 — 1 — 1 — 1 — <—> 1 1 

0.2 0.3 0.4 0.5 0.6 0.7 0.80.9 1 2 

Exposure (Mty) 



Fig. 8. Expected exclusion regions at 90% C.L. as a function 
of the exposure, if no matter effects are observed in MONO- 
LITH. Data have been simulated with Am 2 = +0.003 eV 2 . The 
continuous line shows the expected scaling behaviour, when the 
sensitivity is limited by statistical fluctuations (see text). 



In this study, the systematic uncertainty related to 
the knowledge of the expected neutrino rates has been 
assumed to be 10% up to 400 kty and scaled with the 
square root of the exposure at the other points. Indeed, in 
a real experiment, down-going neutrino events will provide 
a reference sample of "unoscillated" neutrinos to check the 
expected muon neutrinos and anti-neutrinos rates with 
statistically increasing precision. 

The sensitivity to matter effects would be expected to 
improve linearly with the exposure, if it were limited only 
by statistical fluctuations. In the region around the res- 
onance, matter effects determine a v/V asymmetric vari- 
ation in the event rate with respect to pure v^/v T os- 
cillations, that scales as the width of the resonance, i.e. 
sin 26>i3 = V sin 2 26*13. This has to exceed the statistical 
fluctuations in the same region, which decrease as 1/VT, 
where T is the total exposure. Thus the expected exclusion 
limit on sin 2 20i 3 should scale as 1/T. 

As shown in the figure, this scaling law holds up to an 
exposure of around 1 Mty, corresponding to a sensitivity 
around 0.02 on the mixing parameter. Beyond that limit, 
the width of the resonance in the Earth mantle becomes 
comparable to the energy resolution of the experiment and 
the resonance cannot be fully resolved. An improvement 
in the energy resolution would thus be necessary to further 
push the sensitivity to matter effects for sin 2 26>i3 below 
0.01. 

Following the discussion given in [|| , for values of the 
mixing parameter this low, the observation of the reso- 
nance should also be inhibited by other reasons. A size- 
able distortion of the oscillation pattern can be observed 
only when an overlap between the resonance in the effec- 
tive mixing and the oscillation factor occurs. According 
to this argument, in a medium of constant density the ob- 
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servability of the resonance requires a minimum baseline, 
whose length scales as 1/ tan 26*i3 ■ For mixings as small 
as 0.01, this baseline exceeds the Earth's diameter. 



7 Conclusions 

A study of the sensitivity of a massive magnetized detec- 
tor for atmospheric neutrinos (MONOLITH) to matter 
effects in the framework of a three flavour scenario with 
one mass scale dominance for atmospheric neutrinos has 
been reported. 

A sub-dominant component of v e mixing could give 
rise to resonant «-> v e transitions in matter, which occur 
either for neutrinos or for anti-neutrinos only, depending 
on the sign of Am 2 . This can be exploited in a magnetized 
detector to determine the sign of Am 2 and therefore the 
neutrino mass hierarchy. The size of this effect would also 
measure the mixing of electron neutrinos. 

About 200 kty of MONOLITH exposure will be suffi- 
cient to explore regions not yet ruled out by the bound on 
v e — ► oscillations derived by CHOOZ results R). These 
reg ions will be partly accessible to MINOS and JHF |jl6| , 
[jjfl , which, however, can not measure the sign of Am 2 . 

The sensitivity achievable in a high statistics run has 
also been discussed. The ultimate sensitivity which can be 
achieved with exposures of 1-2 Mty would be sin 2 26>i3 ~ 
0.01. In order to go below this limit, the energy resolution 
would have to be improved. For values of sin 2 26>i3 this 
low, the observability of the resonance is also compromised 
by the fact that the maximum of the effective mixing is 
somehow compensated by a minimum in the oscillation 
factor over the Earth's dimension. 
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